feat: send one sample event, response per label set in the configured duration #5298

vamsikrishnakandi · 2024-11-14T10:09:20Z

Description

Send one sample event or response per LabelSet to reporting within the configured duration.

Generate a hash for each report and store it in badger db.
If the hash is already present in the db, do not resend the sample event or response. If the hash is not present, send the sample event/response.
The hash stored in the db expires after the configured TTL.

Linear Ticket

Completes OBS-461

Security

The code changed/added as part of this pull request won't create any security issues with how the software is being used.

…he configured duration

codecov · 2024-11-15T05:49:47Z

Codecov Report

Attention: Patch coverage is 71.79487% with 66 lines in your changes missing coverage. Please review.

Project coverage is 74.76%. Comparing base (7ffac25) to head (b040e13).
Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
enterprise/reporting/reporting.go	46.96%	32 Missing and 3 partials ⚠️
...se/reporting/event_sampler/badger_event_sampler.go	76.04%	19 Missing and 4 partials ⚠️
...nterprise/reporting/event_sampler/event_sampler.go	53.84%	5 Missing and 1 partial ⚠️
...ing/event_sampler/in_memory_cache_event_sampler.go	93.10%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5298      +/-   ##
==========================================
- Coverage   74.78%   74.76%   -0.02%     
==========================================
  Files         433      437       +4     
  Lines       60979    61211     +232     
==========================================
+ Hits        45601    45764     +163     
- Misses      12848    12918      +70     
+ Partials     2530     2529       -1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

…ng to reporting

…into feature/obs-415-support-aggregation-time-as-a-configuration-on-rudder-server

…s/rudder-server into feature/obs-415-support-aggregation-time-as-a-configuration-on-rudder-server

…tion-on-rudder-server' of https://github.com/rudderlabs/rudder-server into feature/obs-415-support-aggregation-time-as-a-configuration-on-rudder-server

docker-compose.yml

scripts/generate-event

…tion-on-rudder-server' into feat.event-sampling-badger

This reverts commit c3d0db9.

…tion-on-rudder-server' into feat.event-sampling-badger

cisse21 · 2024-12-06T09:32:38Z

enterprise/reporting/event_sampler/event_sampler.go

+	case InMemoryCacheTypeEventSampler:
+		es, err = NewInMemoryCacheEventSampler(ttl, eventSamplingCardinality)
+	default:
+		err = fmt.Errorf("invalid event sampler type: %s", eventSamplerType.Load())


probably default to either of the above types and leave a log instead of panicking

cisse21 · 2024-12-06T09:37:31Z

enterprise/reporting/event_sampler/event_sampler.go

+func NewEventSampler(
+	ttl config.ValueLoader[time.Duration],
+	eventSamplerType config.ValueLoader[string],
+	eventSamplingCardinality config.ValueLoader[int],
+	conf *config.Config,
+	log logger.Logger,
+) (EventSampler, error) {
+	var es EventSampler
+	var err error
+
+	switch eventSamplerType.Load() {
+	case BadgerTypeEventSampler:
+		es, err = NewBadgerEventSampler(BadgerEventSamplerPathName, ttl, conf, log)
+	case InMemoryCacheTypeEventSampler:
+		es, err = NewInMemoryCacheEventSampler(ttl, eventSamplingCardinality)
+	default:
+		err = fmt.Errorf("invalid event sampler type: %s", eventSamplerType.Load())
+	}
+
+	if err != nil {
+		return nil, err
+	}
+	return es, nil
+}


Suggested change

func NewEventSampler(

ttl config.ValueLoader[time.Duration],

eventSamplerType config.ValueLoader[string],

eventSamplingCardinality config.ValueLoader[int],

conf *config.Config,

log logger.Logger,

) (EventSampler, error) {

var es EventSampler

var err error

switch eventSamplerType.Load() {

case BadgerTypeEventSampler:

es, err = NewBadgerEventSampler(BadgerEventSamplerPathName, ttl, conf, log)

case InMemoryCacheTypeEventSampler:

es, err = NewInMemoryCacheEventSampler(ttl, eventSamplingCardinality)

default:

err = fmt.Errorf("invalid event sampler type: %s", eventSamplerType.Load())

}

if err != nil {

return nil, err

}

return es, nil

}

func NewEventSampler(

ttl config.ValueLoader[time.Duration],

eventSamplerType config.ValueLoader[string],

eventSamplingCardinality config.ValueLoader[int],

conf *config.Config,

log logger.Logger,

) (es EventSampler, err error) {

switch eventSamplerType.Load() {

case BadgerTypeEventSampler:

es, err = NewBadgerEventSampler(BadgerEventSamplerPathName, ttl, conf, log)

case InMemoryCacheTypeEventSampler:

es, err = NewInMemoryCacheEventSampler(ttl, eventSamplingCardinality)

default:

err = fmt.Errorf("invalid event sampler type: %s", eventSamplerType.Load())

}

return

}

cisse21 · 2024-12-06T09:41:33Z

enterprise/reporting/event_sampler/in_memory_cache_event_sampler.go

+
+type InMemoryCacheEventSampler struct {
+	cache  *cachettl.Cache[string, bool]
+	mu     sync.Mutex


we can get rid of the mu from what I can see.. cachettl already takes care of the safety

cisse21 · 2024-12-06T10:20:24Z

enterprise/reporting/event_sampler/badger_event_sampler.go

@@ -0,0 +1,153 @@
+package event_sampler


add tests for both badger samples and in memory samplers

these are present in event_sampler_test.go

cisse21 · 2024-12-06T10:22:46Z

enterprise/reporting/reporting.go

+
+	if eventSamplingEnabled.Load() {
+		var err error
+		eventSampler, err = event_sampler.NewEventSampler(eventSamplingDuration, eventSamplerType, eventSamplingCardinality, conf, log)


pass ctx into this function and propagate it across

cisse21 · 2024-12-06T10:24:47Z

enterprise/reporting/event_sampler/badger_event_sampler.go

+		return nil, err
+	}
+
+	go es.gcLoop()


use rruntime.Go instead of go

cisse21 · 2024-12-06T10:42:40Z

enterprise/reporting/event_sampler/badger_event_sampler.go

+	})
+}
+
+func (es *BadgerEventSampler) performGC() {


Let's use the same logic we have for dedup and live events

vamsikrishnakandi added 12 commits November 6, 2024 15:33

chore: send multiple reports in a single request to reporting

19e9cfb

Merge branch 'master' into feat.reports-bactching

5c3cab8

chore: default batch size set to 10

1744bbc

chore: added test

6eaa4fd

fix: failing tests

ff63e49

fix: failing tests

98ed866

fix: failing tests

816b213

fix: failing tests

0b926dd

chore: using config from outside

e3185a1

Merge branch 'master' into feat.reports-bactching

676ec7a

Merge branch 'master' into feat.reports-bactching

22a89e8

feat: send sample event and response in one report per label set in t…

3c645f8

…he configured duration

vamsikrishnakandi requested review from ktgowtham, cisse21, itsmihir and satishrudderstack November 14, 2024 10:09

vamsikrishnakandi mentioned this pull request Nov 14, 2024

feat: send one sample event / response per label set to reporting #5289

Closed

1 task

vamsikrishnakandi added 3 commits November 14, 2024 15:47

fix: lint

9e6814f

fix: minor

b22ffc6

fix: minor

bc0acdd

vamsikrishnakandi marked this pull request as ready for review November 15, 2024 06:19

itsmihir and others added 8 commits November 26, 2024 17:06

feat: aggregate reports based on the configured interval before sendi…

e77447d

…ng to reporting

chore: rename aggregationInterval to aggregationIntervalMinutes

7ba6924

feat: modify query to get reports based on aggregation time

a7008e4

Merge branch 'master' of https://github.com/rudderlabs/rudder-server …

bd8c140

…into feature/obs-415-support-aggregation-time-as-a-configuration-on-rudder-server

fix: formatting

57454fe

Merge branch 'feat.reports-bactching' of https://github.com/rudderlab…

36021e0

…s/rudder-server into feature/obs-415-support-aggregation-time-as-a-configuration-on-rudder-server

fix: add aggregation time to the grouping identifiers

58b4f0a

Merge branch 'master' into feat.reports-bactching

308d9ca

Merge branch 'feature/obs-415-support-aggregation-time-as-a-configura…

1e7e49d

…tion-on-rudder-server' of https://github.com/rudderlabs/rudder-server into feature/obs-415-support-aggregation-time-as-a-configuration-on-rudder-server

itsmihir reviewed Dec 6, 2024

View reviewed changes

docker-compose.yml Outdated Show resolved Hide resolved

itsmihir reviewed Dec 6, 2024

View reviewed changes

scripts/generate-event Outdated Show resolved Hide resolved

vamsikrishnakandi and others added 5 commits December 6, 2024 14:19

Merge branch 'feature/obs-415-support-aggregation-time-as-a-configura…

75b6b6c

…tion-on-rudder-server' into feat.event-sampling-badger

fix: tests

8be59d6

fix: revert

1507ca7

Revert "chore: remove bucketStart from getReports and delete query"

11ee199

This reverts commit c3d0db9.

Merge branch 'feature/obs-415-support-aggregation-time-as-a-configura…

06fd221

…tion-on-rudder-server' into feat.event-sampling-badger

Base automatically changed from feature/obs-415-support-aggregation-time-as-a-configuration-on-rudder-server to master December 6, 2024 09:32

cisse21 reviewed Dec 6, 2024

View reviewed changes

chore: merge master

8b3239b

cisse21 reviewed Dec 6, 2024

View reviewed changes

vamsikrishnakandi added 4 commits December 6, 2024 16:22

chore: addressed review comments

8428484

fix: tests

8e51ecc

chore: addressed review comments

fa4f074

Merge branch 'master' into feat.event-sampling-badger

b040e13

vamsikrishnakandi requested review from itsmihir and cisse21 December 9, 2024 05:31

cisse21 approved these changes Dec 9, 2024

View reviewed changes

cisse21 merged commit dc446ee into master Dec 9, 2024
58 checks passed

cisse21 deleted the feat.event-sampling-badger branch December 9, 2024 06:32

This was referenced Dec 9, 2024

chore: release 1.39.0 #5355

Merged

chore: prerelease 1.39.0-rc.1 #5356

Merged

chore: prerelease 1.39.0-rc.2 #5361

Merged

chore: prerelease 1.39.0-rc.3 #5365

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: send one sample event, response per label set in the configured duration #5298

feat: send one sample event, response per label set in the configured duration #5298

vamsikrishnakandi commented Nov 14, 2024 •

edited

Loading

codecov bot commented Nov 15, 2024 •

edited

Loading

cisse21 Dec 6, 2024

cisse21 Dec 6, 2024

cisse21 Dec 6, 2024

cisse21 Dec 6, 2024

vamsikrishnakandi Dec 6, 2024

cisse21 Dec 6, 2024

cisse21 Dec 6, 2024

cisse21 Dec 6, 2024

feat: send one sample event, response per label set in the configured duration #5298

feat: send one sample event, response per label set in the configured duration #5298

Conversation

vamsikrishnakandi commented Nov 14, 2024 • edited Loading

Description

Linear Ticket

Security

codecov bot commented Nov 15, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vamsikrishnakandi commented Nov 14, 2024 •

edited

Loading

codecov bot commented Nov 15, 2024 •

edited

Loading